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Abstract 

In this paper we prove a weak necessary and sufficient maximum principle for stochas- 
tic optimal control problems. Instead of insisting on the maximality condition of the 
Hamiltonian, we show that belongs to the sum of Clarke's generalized gradient of the 
Hamiltonian and Clarke's normal cone of the control constraint set at the optimal con- 
trol. Under a joint concavity condition on the Hamiltonian and a convexity condition on 
the terminal objective function, the necessary condition becomes sufficient. We give two 
examples to demonstrate the weak stochastic maximum principle. 
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1 Introduction 

There has been extensive research in the stochastic control theory. Two principal and 
most commonly used methods in solving stochastic optimal control problems are the dy- 
namic programming principle and the stochastic maximum principle (SMP). The books by 
Fleming- Rishel |12j . Fleming-Soner [13], and Yong-Zhou [23J provide excellent expositions 
and rigorous treatment of the subject of the dynamic programming principle in the optimal 
deterministic and stochastic control theory. 

Many people have made great contributions in the research of the SMP. Kushner |17[ 
[T8] is the first to study the necessary SMP. Haussmann [15] . Bensoussan [2] and Bismut 
[31 HI [S] extend Kushner's SMP to more general stochastic control problems with control- free 
diffusion coefficients. Peng [20] applies the second order spike variation technique to derive 
the necessary SMP to stochastic control problems with controlled diffusion coefficients. Zhou 
[25j simplifies Peng's proof. Cadenillas-Karatzas [6] extends Peng's SMP to systems with 
random coefficients and Tang-Li [22J with jump diffusions. Bismut [5] is the first to investigate 
the sufficient SMP. Zhou [26J proves that Peng's SMP is also sufficient in the presence of 
certain convexity condition. Framstad-0ksendal-Sulem p3] extends the sufficient SMP to 
systems with jump diffusion, Donnelly [11] with Markovian regime-switching diffusion and, 
most recently, Zhang-Elliott-Siu [24J with Markovian regime-switching jump diffusion. 

Briefly speaking, the necessary SMP states that any optimal control along with the op- 
timal state trajectory must solve a system of forward-backward stochastic differential equa- 
tions plus a maximum condition of the optimal control on the Hamiltonian. The necessary 
condition together with certain concavity conditions on the Hamiltonian give the sufficient 
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condition of optimality. The major difficulty of generalizing the classical Pontryagin's maxi- 
mum principle to a stochastic control problem with controlled diffusion term is that, in some 
cases, the Hamiltonian is a convex function of the control variable and achieves the minimum 
at the optimal control (see |23[ Example 3.3.1]). One of the major contributions of Peng's 
SMP is the introduction of the generalized Hamiltonian and the second order adjoint stochas- 
tic processes. In those cases where the Hamiltonian is convex, it is the second order term 
that turns the generalized Hamiltonian to a concave function which achieves the maximum 
at the optimal control. The generalized Hamiltonian and the second order adjoint equation 
are introduced to preserve the maximality condition of Pontryagin's maximum principle. 

However, the second order terms also pose problems. Firstly, one has to assume that 
all functions involved are twice continuously differentiable in the state variable in order 
to use the second order variation, which limits the scope of problems applicable to the 
theorem. Secondly, one has to solve the associated second order adjoint backward stochastic 
differential equation (BSDE) with the dimensionality equal to the square of that of its first 
order counterpart, which makes the problem more difficult to solve, at least numerically. 
Lastly, one can not get the sufficient condition by enhancing the necessary condition with 
some joint concavity condition to the generalized Hamiltonian and instead one has to add 
some joint concavity condition to the Hamiltonian (compare [231 Theorem 3.3.2] and [23} 
Theorem 3.5.2]), which illustrates that the necessary SMP is not completely compatible 
with the sufficient SMP. This motivates us to relax the requirement of the maximality of 
the Hamiltonian at the optimal control and to seek a weak but compatible necessary and 
sufficient SMP. 

The main contribution of this paper is that we prove a weak version of the necessary and 
sufficient SMP. Instead of insisting on the Hamiltonian to achieve the maximum at the opti- 
mal control, which is in general impossible, we relax the necessary condition by only requiring 
the optimal control to be a stationary point of the Hamiltonian. Specifically, we prove that 
belongs to the sum of Clarke's generalized gradient of the Hamiltonian and Clarke's normal 
cone of the control constraint set at the optimal control almost surely almost everywhere. 
Under the joint concavity condition on the Hamiltonian and the convexity condition on the 
terminal objective function, the necessary condition becomes the sufficient condition. The 
advantage of the weak SMP is the following. Firstly, the second order differentiability of the 
coefficients and the objective functions in the state variable is not required as the weak SMP 
does not have any second order terms. Secondly, the dimensionality of the BSDE is much 
reduced as the second order adjoint process is not involved. Lastly, the necessary condition 
and the sufficient condition are compatible with each other in the sense that the necessary 
condition provides a stationary point while the sufficient condition confirms its optimality, 
which is in the same sprit as the necessary and sufficient conditions in the finite dimensional 
optimization. 

The rest of the paper is organized as follows. Section 2 introduces the notations, the 
formulation of the stochastic control problem and the basic assumptions. Section 3 recalls 
some basic concepts of Clarke's generalized gradient and normal cone which are needed for 
the main results. Section 4 states the main theorems of the paper and gives two examples to 
demonstrate the power of the weak SMP. Section 5 establishes some useful preliminary results 
on moment estimates, Lipschitz property, Taylor expansion and duality analysis. Section 6 
proves the main theorems (Theorems 14.11 and I4.2p . Section 7 concludes. 

2 Problem Formulation 

In this section, we formulate the stochastic control problem and introduce some assumptions. 
We start with a brief summary of the notations that will be used throughout the paper. 
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Let (fi, {Tt}t>o, P) be a complete filtered probability space. Suppose that J-"o contains 
all the P-null sets in T and {Ft}t>o is the right continuous natural filtration generated 
by W(t), an m-dimensional standard Brownian motion, augmented by all the P-null sets 
in T . We consider a stochastic control model where the state of the system is governed by 
a controlled stochastic differential equation 



dx(t) = b(t, x(t),u(t))dt + a(t, x(t),u(t))dW(t), t G [0, T], 
x(Q) = x 



(1) 



where u is a R k valued progressively measurable (with respect to {J-t}t>o) process, and T > 
is a fixed finite time horizon, b : [0, T] x R n x R k -> R n , cr = (a 1 , . . . , a™) : [0, T] x W l x M fc -> 
(M n ) m are given functions satisfying the following assumptions: 

(Al) The maps b and a are measurable, and there exists constants L > 0, K > such that 
for (p = b and a we have 

\4>(t, x, u) — 4>(t, x,u)\ < L\x — x\ + K\u — u\, 
Vt € [0,T], x,x G M n , u,« G R k , 
\4>(t,Q,u)\ <L + K, V(t,n) G [0,T] x 

(A2) The maps b and cr are C 1 in x. Moreover, there exist constants L > and a modulus 
of continuity uj : [0, +oo) — > [0, +oo) such that for (j) = b and a, we have 

|<^ x (t, a;, u) — (f> x (t, x, u)\ < L\x — x\ + uj(d(u, u)), 
Vt G [0,T], x, x G M n , u,u G M fc . 



Consider the cost functional 



J(u) = S 



[ T f(t,x(t),u(t))dt + h(x(T)) 
Jo 



(2) 



where / : [0, T] x 
assumptions: 



and h : 



are given functions satisfying the following 



(A3) The maps / and h are measurable and there exists constants Ki, > such that 
\f(t,x,u) - f(t,x,u)\ < {K 1 + K 2 (\x\ + \u\ + \u\))\u-u\. 

(A4) The maps / and h are C 1 in x. Moreover, there exist constants L > and a modulus 
of continuity O : [0, +oo) — >■ [0, +oo) such that for <\> = f and h, we have 

\4> x (t, x, u) — 4> x (t, x,u)\ < L\x — x| + uj(d(u, u)), 
Vt G [0,T], x, x G W\ u,u£R k 
\<l> x (t,0,u)\ <L + K, V(t,u) G [0,T] x R k . 

Remark 2.1. Assumptions (A3) and (A4) together cover many cases, including all quadratic 
functions in x and u. For instance, if / is Lipschitz in u, then K 2 = 0. On the other hand, if 
/ is differentiable with respect to u and /„ satisfies a linear growth condition in u, then K 2 
is a positive constant. 

A control u is called admissible if it is a progressively measurable process, valued in U, a 
nonempty closed convex subset of R h , such that 



E\u(t)\ 4 dt) < oo. 



(3) 
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Denote by U a d the set of all admissible controls. Consider a measure space (S, S, //), where 
S = O x [0, T], £ denotes the predictable <r-field generated by the Ft adapted Brownian 
motion and /i = Px Leb. Define L 4 (5; R fc ) to be the Banach space of R fc valued £ measurable 
functions / such that the norm is defined by Q . Since £ is a predictable cr-field we have that 
/ € L 4 (S;M. k ) is progressively measurable. Under assumption (Al), for any u E L 4 (S;M. k ), 
the state equation (pQ) admits a unique solution, and the cost functional ([2]) is well defined. 
In the case that x is a solution of (pQ) corresponding to an admissible control u € U aa \, we 
call (x, u) an admissible pair, and x an admissible state process. 
Our optimal control problem can be stated as follows 

Problem (S) Minimize © over M a d- 

Any u £ ^/ a d satisfying 

J(u) = inf J(u) 

u&U ad 

is called an optimal control. The corresponding x and (x, are called an optimal state 
process and optimal pair, respectively. 

3 Clarke's Generalized Gradient and Normal Cone 

In this section we recall some basic concepts and properties in nonsmooth analysis and 
optimization, which are needed in the statement and proof of the main results (Theorems 
14.11 and 14. 2\) . Clarke's generalized gradient is first introduced to the finite dimensional space 
in [7] and then extended to the infinite dimensional space in [HI [9] and [1] . Interested readers 
may refer to [10] for a detailed and complete treatment of the topic. 

Definition 3.1. (Generalized directional derivative) Let C be an open subset of a Banach 
space X, and let a function f : C — > R be given. We suppose that f is Lipschitz on C. The 
generalized directional derivative of f at x in the direction v, denoted f°(x;v), is given by 

f (x;v) = hmsup . 

Definition 3.2. (Clarke's generalized gradient) Let X* denote the dual of X and (•,•) be 
the duality pairing between X and X* . The generalized gradient of f at x, denoted df(x), is 
the set of all £ in X* satisfying 

f°(x;v) > (v,C) forVv £ X. 
Theorem 3.3. If f attains a local minimum or maximum at x, then S df(x). 

Theorem 13.31 is only valid in the case where C is open. When the function is defined on 
a general non-empty subset of X, we need to introduce the so-called distance function and 
the concept of Clarke's tangent cone and normal cone. 

Definition 3.4. (Distance function) Let X be a Banach space and C be a non-empty subset 
of X. The distance function dc '■ X — > R is defined as 

dc{x) = inf { || x — c\\ : c £ C}. 

Theorem 3.5. The function dc satisfies the following global Lipschitz condition on X 

\d c {x) - d c {y)\ < \\x - 2/|| - 



4 



Definition 3.6. (Adjacent cone) Let C be the closure of C and x E C. The adjacent cone 
to C at x, denoted as Tq(x), is defined by 

T%(x):={v\ lim d C (x + hv)/h = 0}. 

h->0+ 

Definition 3.7. (Tangent cone) Suppose x € C . A vector v in X is a tangent to C at 
x provided d^(x;v) = 0. The tangent cone to C at x, denoted as Tq(x), is the set of all 
tangents to C at x. 

In addition, when the set C is convex, it can be proved that the adjacent and tangent 
cones coincide, see [U Proposition 4.2.1]. 

Theorem 3.8. Assume that C is convex. Then Tc(x) = T^(x). 

Definition 3.9. (Normal cone) Let x € C . The normal cone to C at x is defined by the 
polarity with Tc(x): 

N c (x) = {£ € X* : (f,u) < for all v G T c {x)}. 
The following necessary optimality condition is proved in [101 page 52 Corollary]. 

Theorem 3.10. Assume that f is Lipschitz near x and attains a minimum over C at x. Then 
0edf(x) + N c (x). 

4 Weak Stochastic Maximum Principle 

In this section we state the weak necessary and sufficient stochastic maximum principles and 
give two examples to illustrate the main results. 

The Hamiltonian H : [0,T] x R n x R k x R n x R nxm — > R, for the stochastic control 
problem ([1]) and (|2|), is defined by 

H(t,x,u,p,q) = (p,b(t,x,u)} +tr[q T a(t,x,u)} - f(t,x,u). (4) 

Given an admissible pair (x,u), the adjoint equation is the following linear BSDE: 

m 

dp{t) = -{b x (t,x(t),u(t)) T p(t) + Y / 4(t^(t),u(t)fq j (t) 

3=1 (5) 

- f x (t,x(t),u(t))}dt + q(t)dW(t), t€ [0,7] 
L P (T) = —h x (x(T)). 



Substitute the Hamiltonian H defined in we have 

dp{t) = -H x (t,x(t),u(t),p(t),q(t))dt + q(t)dW(t), t € [0, T] 
p(T) =-h x (x(T)). 



(6) 



Any pair of stochastic processes (p, q) € Ljr(0,T;M n ) x (L^O, T; R n ) m ) is called an adapted 
solution of (JSJ), where q = (# ,... ,q m ) and Lj-(0,T;R n ) is the space of R n - valued square 
integrable progressively measurable processes. Existence and uniqueness of the solutions of 
BSDEs driven by Brownian motions have been extensively studied in |19j . According to their 
result, we know that under assumptions (A1)-(A4), for any (x,u) € Ljr(0,T;R n ) xU a d, © 
admits a unique adapted solution (p,q). If (x,u) is an optimal (resp. admissible) pair, and 
(p,q) is the adapted solution of ©, then (x,u,p,q) is called an optimal (resp. admissible) 
4-tuple. 

We can now state the main results of the paper. 
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Theorem 4.1. (Weak Necessary SMP) Let assumptions (A1)-(A4) hold. Let (x,u) be 
an optimal pair of Problem (S). Then there exists a pair of stochastic processes (p,q) G 
L^(0,T;R n ) x (L3r(0,T;]R n ) m ) satisfying the adjoint equation such that 

G d u {-H){t,x(t),u(t),p(t),q(t)) + Nu(u{t)), a.e. t G [0,T], P-o.s. (7) 

Theorem 4.2. (Weak Sufficient SMP) Let assumptions (A1)-(A4) hold and let (x,u,p,q) 
be an admissible 4-tuple satisfying Suppose further that h is convex and H(t, ■, -,p(t),q(t)) 
is concave for all t G [0,T] a.s. Then (x,u) is an optimal pair for Problem (S). 



We consider two examples from |23| and derive the same results as those in [23] using 
Theorem 14.11 and Theorem 14.21 A key property to use in our approach is the adaptedness of 
the adjoint process. 



dx(t) = u(t)dW(t),t G [0,1] 
x(0) = 



Example 4.3. Consider the following stochastic control problem [231 Example 3.5.3]: 

(8) 

with the control constraint set U = [0, 1] and the cost functional 

J(u) = E^-J^ u{t)dt + ^(l) 2 }- 
Suppose (x, u) is an optimal pair, then the corresponding adjoint equation is 



(9) 



dp(t) = q(t)dW(t), t G [0,1] 
p(l) = -x(l). 

Using (JSJ) and @ and via a simple calculation we obtain 

p(t) = - I u(s)dW(s)- [ (u(s) + q(s))dW{s). 
Jo Jt 

Since the adjoint process p(t) is adapted to the filtration Ft , we must have 

u(t) + q(t) = for all t G [0, 1], F-a.s. (10) 

The corresponding Hamiltonian is 

H(t, x, u,p(t),q(t)) = q(t)u + u. 

Since the problem satisfies (A1)-(A4), by Theorem 14.11 and ©, we have 

G -(q(t) + 1) + N m {u(t)) for all t G [0, l],P-o.a. 

Consequently, on any nonzero measurable set £ £ S = fl x [0,1], we can only have the 
following three cases: 

Case 1 : < u(t) < 1 JV [0j i] (u(t)) = {0} and q(t) = -1. 
Case 2 : u(t) = =>- N [0A] (u(t)) = (-00, 0] and q(t) + 1 < 0. 
Case 3 : u(t) = 1 N [0A] (u(t)) = [0, +00) and q{t) + 1 > 0. 



6 



Suppose Case 1 or Case 2 is true, then u(t) + q[t) < u(t) — 1 < for some nonzero measur- 
able set E G S, contradiction to (|10p . Hence, we have u(t) = 1 for every t G [0,1], P- 
a.s. and x(t) = W(t) and (p(t),q(t)) = (-W(t),-l) for t G [0,1]. Since (x,u) i-> 
H(t,x,u,p(t),q(t)) = —u + u = is concave and x i-> = ^x 2 is convex, we conclude 
that u(t) = 1 is the optimal control using Theorem 14.21 

Example 4.4. Consider the following stochastic control problem [23\ Example 3.3.1]: 

/ dx(t)=u(t)dW(t),t£ [0,1] 

\ x(0) = { } 

with the control constraint set U = [—1,1] and the cost functional 



[x(t) 2 - ^u{t) 2 ]dt + x{l) 2 



(12) 



Suppose (x, u) is an optimal pair, then the corresponding adjoint equation is 

f dp(t) = 2x(t)dt + q(t)dW(t),t G [0, 1] 
\ p(l) = -25(1). 

Using (|lip . (|12p and via a simple calculation, we obtain 

p(t) = -2 / (2 - *)«(s)(flV(a) - / ((4-2s)u(s) + g(s))dW(s). 
Jo Jt 

Since the adjoint process p(t) is adapted to the filtration Ft, we must have 

(4 - 2t)u{t) + q{t) = for all t G [0, 1], F-a.s. (13) 
The corresponding Hamiltonian is 

H{t, x, u,p(t), q(t)) = q(t)u - x 2 + ^u 2 . 

Since the problem satisfies assumptions (A1)-(A4), by Theorem 14. II and ([7|. we have 

G - (q(t) + u(t)) + N { _ 1A] (u(t)) for all t G [0,l],P-o.a. 

Consequently, on any nonzero measurable set E G S, we can only have the following three 
cases: 

Case 1 -1 < u(t) <1=> N { _ hl] (u(t)) = {0} and q(t) + u(t) = 0. 
Case 2 u{t) = -I N[_ ltl ](u(t)) = (-oo,0] and q(t) + u(t) < 0. 
Case 3 u(t) = 1 N[_ 1A] (u(t)) = [0, +oo) and q(t) + u(t) > 0. 
Suppose Case 2 were true. Then 

(4 - 2t)u(t) + q(t) < -3 + 2t < -1. 
Contradiction to (fT3|) . Suppose Case 3 were true. Then 

(4 - 2t)u(t) + q(t) > 3 - 2t > 1. 
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Contradiction to (fT3"j) . Hence we have 



-1 < u(t) < 1 and q(t) + u(t) = for all t G [0, l],¥-a.s. 

By (USD, (HU) and (JT2J) , we obtain (x(t),u(t)) = (0,0) and (p(i),g(i)) = (0,0). However, 
since (x,u) h-> H(t,x,u,p(t),q(t)) = —x 2 + ^-u 2 is not a concave function, Theorem 14,2 
cannot be used. We have to use other method to check the optimality of u. Substituting 
x(t) = Jq u(s)dW(s) into the cost functional and via simple calculations, we obtain 



J(u) 



£ /'(H" (()2 " f 



Hence, we conclude that u(t) = is indeed the optimal control. 

Remark 4.5. Note that our approach in Example 14.41 is different from that in [23, Example 
3.3.1] in the following sense. Instead of trying to find the optimal control directly from the 
objective function, we first find all admissible controls satisfying the necessary condition 
stated in Theorem 14.11 After that, by using the adaptedness property of the adjoint process, 
we prove that if an optimal control exists, then it must be u(t) = 0. Finally, by calculating 
the cost function explicitly, we verify that our candidate u(t) = is indeed the optimal 
control. Note also that the Hamiltonian u h-> H(t,x(t),u,p(t),q(t)) = ^v 2 is a convex 
function and u(t) = is a minimum point. This is the reason that [23] introduces the 
generalized Hamiltonian T~L which makes u(t) = a maximum point. 

5 Preliminary Results 

In this section, we prove some preliminary results, which will be useful in the sequel. Here- 
after, K represents a generic constant. 

5.1 A Moment Estimate 

The following result is a simplified version of [23, Lemma 3.4.2]. 
Lemma 5.1. Let Y(t) G 1^(0, T; W 1 ) be the solution of the following: 

m 

dY(t) = {A(t)Y(t) + a(t)}dt + ^{S J '(t)y(t) + p j (t)}dW j (t), 

{ no) =y , 



where A, : [0,T] X Q — > R nxn and a,f3 j :[0,T]xfi — >■ W 1 are {F t }t>o-adapted, and 

(14) 



\A(t)\, \Bi(t)\ < L, a.e.te[0,T], P-a.s., 1 < j < m, 
r T i-T 



[ E\a(s)\ 2k ds+ [ E\p j {s)\ 2k ds < oo, 1 <j < m, 
Jo Jo 



for some k > 1. Then 



sup E\Y{t)\ 2k < K < 
te[o,T] 



E\Y \ 2 k + £ E\a{s)\ 2k ds + Y^j E\(3 j (s)\ 2k ds I . (15) 
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Proof. For notation simplicity, we prove only the case m = 1, leaving the case m > 1 to the 
interested reader. We first assume that a and /3 are bounded. Let e > and define 



{Y) e 4 Vl^| 2 + e 2 , V Y G M n . 



(16) 



Note that for any e > 0, the map y i— > (y) e is smooth and (Y) e — > \Y\ as e — > 0. Applying 
Ito's fomula to (Y(t)) 2k , we have 



d(y(t))f = 2A:(y(t))f- 1 ^ra- 



{[A(i)Y(t) + a(i)] dt + [B(t)Y(t) + (3(t)} dW{t)} 



+ 



fc(2A; - l)(Y(t))f- 2 ^^ + fc(Y(t))f- 1 e ' 



<y(*)>i 



[B(t)Y(t) + f3(t)] 2 dt. 



(Y(t)) 2 

Since (Y(i)) e > |Y(i)| and 2fc - 1 > 1, we obtain 

d{Y(t))f < 2k(Y(t))f- 1 {[A(t)Y(t) + a(t)]dt + [B(t)Y(t) + P(t)]dW(t)} 



+ 



[B(t)Y(t) + (3(t)] 2 dt. 



k{2k-l){Y{t))f- 2 
Writing it in the integral form and taking the expectation, we have 

E(Y(t)) 2k < E(Y(0)) 2k + 2kE Ay(s))f _1 {|A(s)|(Y(s)) e + \a(s)\}ds 

Jo 

+ m - i)e Ay( s )) 2fe - 2 {|i?(,)|(y( s )) £ + W(s)\} 2 ds 

Jo 

< E(Y(0)) 2 e k + KaEj* {(Y(s)) 2 e k + WsWis))?- 1 

+ms)\ 2 (Y( S )) 2k - 2 }ds, 

where Kq is a constant independent of t. Applying Young's inequality, we get 

E(Y(t)) 2k < E(Y(0)) 2k + KE J*{(Y(s)) 2k + \a(s)\ 2k + |/3(s)| 2fc } ds. 
Finally, Gronwall's inequality yields 



sup E(Y(t))f < K \ £(Y(0))f + E 
te[o,T] I ./o 



*{s)\ 2k + \P{s)\ 



2k 



ds 



for some constant K = K(L, k,T). Letting e — >■ in (fTUj) . then (JT7j) becomes (fT5j) . 
5.2 Lipschitz Property 



(17) 
□ 



Lemma 5.2. Lei ui,U2 € £ (<f?;R ) anc? xi,£2 &e the associated state processes satisfying 
([T]) . Then we have the following inequality: 

sup E\x\{t) — X2(i)| 4 < K ||m — 1*2 1| 

iG[0,T] 

Proof. Let = — X2(i). Then we have 

de(t) = [b(t,x 1 (t),u 1 (t)) -b(t,x 2 (t),u 2 (t))]dt 

+ [a(t, Xl (t), Ul {t)) - a(t,x 2 (t),u 2 (t))]dW{t), 



b x (t)S(t) + b(t, x 2 (t), Ul (t)) - b(t, x 2 (t),u 2 (t)) 



dt 



+ + x 2 (t), «i(t)) - a(t, x 2 (t),u 2 (t))] dW(t), 
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where for <f> = b and a, 

4>x(t)= [ <l>x(t,X 2 (t)+e(x 1 (t)-X 2 (t)),Ul{t))dB. 

Jo 

By Lemma |5.1( we obtain 

sup E\i{t)\ A <K\! E\b(s,X2(s), Ul (s))-b(s,X2(s),u 2 (s))\ 4 ds 

tg[0,T] I JO 

f T 

+ / E\a(s,X2(s),ui(s)) - a(s,X2{s),U2{s))\ 4 ds 
Jo 

<k{J q E[K A \ Ul (s)-U2(s)\ A ]ds 

<K\\ Ul -U2\\ A . 



□ 



Lemma 5.3. Assume that (Al) holds. Given control u € L 4 (5;IR fc ) and associated state 
process x satisfying ([!]), then 



E\ sup \x(t)\ 2 ) <k(i + \x \ 2 + [ E\u(t)\ 2 dt) 

\0<t<T J \ Jo J 



(18) 



Proof. For every integer k > 0, define the stopping time 

r fc = T A inf {t € [0, T] : \x(t)\ > k}. 

Clearly, r k \ T a.s. Set x k (t) = x(tArk) and itfc(t) = w(i Art) for i G [0,T]. Then x k (t),u k (t) 
satisfy the equation 

x k (t)=x + / 6(s,a;jb(s),wjfc(s))J[o )7>e ](s)ds+ / a(s,Xk{s),u k (s))I[ 0>Tk \(s)dW(s). 
Jo Jo 

Using the elementary inequality \a + b + c\ < 3(\a\ 2 + \b\ 2 + |c| 2 ), Holder's inequality and 
(Al), we have 

\x k (t)\ 2 <K\x \ 2 + Kt f (1 + \x k (s)\ 2 + \u k (s)\ 2 ) ds 
Jo 

ft 2 

+ K 



a(s,x k (s),u k (s))I [0jTk] (s)dW(s) . 
By Doob's martingale inequality (see [16]) and (Al), we have 

e( sup Ms)! 2 ) <K|x | 2 + KT (\l + E\x k {s)\ 2 +E\u k {s)\ 2 )ds 

\0<s<t J Jo 

+ AKE [ \a(s,x k (s),u k (s))\ 2 I [0tTk] (s)ds 



<K |l + |x | 2 + jf £K(s)| 2 + E\x k (s)\ 2 ds^ 



Gronwall's inequality yields that 



E\ sup \x k {t)\ 2 \ <k(\ + \x \ 2 + [ E\ Uk (t)\ 2 dt^ 

\0<t<T J \ Jo J 

Finally, the required inequality (|18p follows by letting k — > oo. □ 
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Lemma 5.4. The cost functional J : L 4 (S;M. k ) — > R is locally Lipschitz, i.e., for all u in 
L A (S;M. k ) there exists a small ball B~~ with radius M > containing u on which , we have 

| J («i) - J (u 2 ) | < K Mt a\\ux - u 2 \\ (19) 

for Vui,«a 6 5f . 

Proo/. Given u € L 4 (5;M fc ) and M > 0, define 

= |u g L 4 (5; M fc ) : ||u - u|| < m} . 

For any u\,u 2 € with associated state process x\,x 2 , according to (A4) we have 
E\h( Xl (T)) - h(x 2 (T))\ 

<E [ \(h x (xi(T) + 6(x 2 (T) - Xl (T))) ,x 2 (T) - Xl (T))\de 



o 



< K{E (1 + |xx(T)| 2 + |x 2 (T)| 2 )}^ {£|x 2 (T) - Xl (T)\ 2 Y 

<K{E(l + \x(T)\ 2 + \x(T) - xi(T)| 2 + \x(T) - x 2 (T)\ 2 ) p {E\x 2 (T) - Xl (T)\ 2 y 
by Holder's inequality and Minkowski's inequality. According to Lemma 15.3} Jensen's in- 



equality and Lemma 15,2 



E\h{ Xl {T))-h{x 2 {T))\<K M \ E\u(t)\ 2 dtY \ {E\x 2 (T) - Xl (T)\ 2 y- 



<K M {l+(^£ E\u(t)\ 4 dty } {E\x 2 (T) - X!(T)| 4 } 3 
< K M ,u\\ui -u 2 \\. 

Similarly, we have 

E [ \f(t, Xl (t), Ul (t))- f(t,x 2 (t),u 2 (t))\dt 
Jo 

<E [ \f(t, Xl (t), Ul (t)) - f(t,x 2 (t), Ul (t))\ + \f(t,x 2 (t), Ul (t)) - f(t,x 2 (t),u 2 (t))\dt. 
Jo 

Following similar arguments, we have 

E [ \f(t, Xl (t), Ul (t))- f(t,x 2 (t), Ul {t))\dt<K M ,4 Ul -u 2 \\. 
Jo 

For the second term, by (A3) we have 

E [ \f(t,x 2 (t), Ul (t)) - f(t,x 2 (t),u 2 (t))\dt 
Jo 

< E I {K l + K 2 {\x 2 {t)\ + \ui(t)\ + \u 2 (t)\)} \m{t) - u 2 (t)\dt 
Jo 

<k\e ( (1 + \x 2 (t)\ 2 + \ Ul (t)\ 2 + \u 2 (t)\ 2 ) dtV — t*2|| 



<K M ( l + |^^ T |«(t)| 4 | 4 J ||U1-U2| 



< K M ,u\\ui ~u 2 \\. 

19|) follows by combining the above inequalities. □ 
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5.3 Taylor Expansions 

Let (x,u) be an admissible pair. Let v 6 L 4 (5;M fc ) and e > 0. Define u e (t) = u{t) + ev{t) for 
all t G [0, T]. Let (x e ,u e ) satisfy the following stochastic control system: 

dx e (t) = b(t,x e (t),u e (t))dt + a(t,x e (t),u e (t))dW(t), te[0,T], 
x e (0) = xo- 

Next, for tp = b, a 3 , (1 < j < m), f, we define 

Vx(t) = ip x (t, x(t),u(t)), 6(p(t) = <p(t, x(t),u e (t)) - cp(t, x(t),u(t)). 

Let y e be the solution of the following stochastic differential equation: 



dy e (t) = {b x (t)y*(t) + Sb(t)} dt + E {<d{t)tf{t) + 6a'(t)}dW3{t), te[0,T], 

3=1 V } 

I y e (o) =o 



(20) 



Remark 5.5. The variation in our proof is different from the so-called spike variation tech- 
nique in the proof of Peng's maximum principle in [20] and [23]. In their proof, where 
u e {t) = u(t) + h TT + e ]v(t), one first perturbs an optimal control on a small set of size e and 
then let e — > 0. Whereas, in our proof we perturbs an optimal control over the whole space. 
Then reason behind this is that in the definition of Clarke's generalized directional deriva- 
tive, v(t) represents a directional vector in L 4 (S';]R fc ) and must be fixed. One perturbs the 
control through multiplication of a scalar e and letting e — > 0. 

The following lemma gives the Taylor expansion result of the state process and cost 
functional. 

Lemma 5.6. Let assumptions (A1)-(A4) hold. Then, we have 

sup Elx'it) -x(t)\ 2 = 0(e 2 ), (21) 
te[o,T] 

sup E\y*(t)\ 2 = 0(e 2 ), (22) 

*€[0,T] 

sup E\x e (t)-x(t)-y e (t)\ 2 = o(e 2 ). (23) 
te[o,T] 

Moreover, the following expansion holds for the cost functional: 



J(« e )= J(u)+E{h x (x(T)),y%t)) 

+E [ {(Mt,x(t),u(t)),y £ (t)) + Sf(t)} dt + o(e) 



(24) 



o 



Proof. For simplicity, we carry out the proof only for the case n = m = 1. 

Proof of ([21]) • Let £ e (i) = x e (i) -x(t). The we have 

' d^it) ={fr x (t)?(t) + 6b(t)}dt 

+ {o£(t)£ e (t) + Sa(t)} dW{t), t E [0, T], (25) 

, e(o) =0. 

where for <j> = b and a, 

k(t)= [ Mt,x(t)+9(x*(t)-x(t)),u e (t))de. (26) 



n 



12 



By Lemma I5TTI since b %(t) and 5%{t) are bounded according to assumption (Al), we obtain 

sup E\C{t)\ 2 <K [ E{\5b(s)\ 2 + \5a(s)\ 2 }ds 
te[o,T] Jo 

r T 

< Ke 2 / E\v(s)\ 2 ds 

Jo 

< Ke 2 . 

This proves (|2Tj) . 

Proof of ([22j) . Similarly, b x (t) and cr x (t) are bounded according to assumption (Al). Ap- 
plying Lemma 15, II to (|20p . we obtain 

sup E\y e (t)\ 2 <K [ E{\5b{s)\ 2 + \5a(s)\ 2 }ds < Ke 2 . 
te[o,T] jo 



This proves 



Proof of d23j). Let C e (i) = x e {t) - x(t) - y e (t) = f(t) - y e (t). Then, by ([25]) and ((20]) we 
have 

dC(t) =dC(t) - dy e (t) 

= {K(t)e(t) - b x (t)y e (t)} dt + {aUt)C(t) - a x (t)y e (t)} dW(t) 
= {b x (t)C(t)+[b x (t)-b x (t)] y*{t)}dt 

+ mt)c(t) + mt) - a x {t)\ f(t)} dw(t) 

Since b x (t) and a x (t) are bounded by assumption (Al), applying Lemma I5TT1 we obtain 



su Pte[0 ,T] E\C(t)\ 2 < k E^[b%{t) - b x (t)] v e (t)f + W x (t) - <r*(t)] v e (t)\ 2 } 



dt. 



(27) 

By Holder's inequality, (|26|) . (|22p and ([2ip . also note that for any p > 0, there exists a 
constant K p > such that u>(r) < p + rK p for all r > 0, we have 



< 



b%(t)-b x (t) y e (t) 



dt 



E 



i 

4\ 2 



b%{t)-b x {t) ) [E\tf{t)\ A Ydt 



< {El \b x {t,x{t)+9e{t),u e {t))-b x {t)\d0\ e l dt 



T 



<K {E(C(tf + uj{ev(t)f)ye 2 dt 
Jo 

<K [ {e 4 + E[p + K p e\v(t)\fy e 2 dt 



<K\e 2 + p 2 + K 2 e 2 {E\v(t)\ 4 y dt e 2 . 
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Hence the first term in (|27p is o(e 2 ), the second term can be shown similarly being o(e^ 
which gives ()23p . 



Proof of (|24p . By definition of the cost functional ([2]), we have 
J(u £ ) - J(u) 

=E {h(x*(T)) - h(x(T))} + E ^ {fit, x%t),u%t)) - f(t, x(t),u(t))} dt (28) 



JO 

For the first term on the right side of (j28j) we have 
E {h(x e (T)) — h(x(T))} 

= e I (h x (x(T) + ee(T)),e(T))de 

Jo 

= E(h x (x(T)),y*(T)) + E(h x (x(T)),C(T)) 

+ E [ {h x {x{T) + eC{T))-h x {x{T)),C{T))de. 
Jo 

Then, by (|2T|) . ([23]) . (A4) and applying Holder's inequality, we have 

E {h(x*(T)) - h(x(T))} = E(h x (x(T)),y*(T)) + o(e). (29) 
For the second term on the right side of (j28|) we have 

E f {fit, ar 6 (t), « 6 (t)) - fit, x(t),u(t))} dt 
Jo 

=e J U (f x (t,x(t) + e?(t),u e (t)),e(t))de 

+ {f{t, x{t),u e (t)) - f{t, x{t),u{t))} dt 

=E [ {(f x it,xit),uit)),y*it)} + 5fit)} 
Jo 

+ H (f x {t, xit) + 6Cit), n e (t)) - f x {t, xit),uit)),y e it))d6 

+ U (f x {t,xit) + ee(t),u e (t)),C(t))de\ dt 

Then, using (A4) and by a similar argument as in the proof of (j23[) . we have 

E [ T {fit, x^t),u^t)) - fit, x(t),u(t))} ^ 

-T 







(30) 

= E I {(f x it,xit),uit)),y'it)) + 5fit)} + oie). 
Jo 

21 follows from (ESL and W). □ 



5.4 Duality Analysis 

Lemma 5.7. Let assumptions (A1)-(A4) hold. Let y e be the solution of (|20p and (p,q) be 
the adapted solution of ([S]). Then 

E(piT),y\T)) = E [ T {<p(t),5&(t)> + (/*(*) , W + tr [qitf 5ait)]} dt (31) 
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o 



Proof. Applying Ito's lemma we get 



d( P (t),y t (t)) = { 



( P (t),b x (t)y e (t) + Sb(t)) + ( g(t), £ H(t)y e (t) + 5o*(tj\ 



3=1 



+ 



,y e {t)))dt 



+ Up(t),J^[4(t)ym + ^(t)]j + (q(t),y*(t)) } dW(t). 
Writing it in integral form and taking the expectation we have 



E(p(T),y e (T)) = E f < 
Jo 



(p(t),Sb(t)) + </*(*), j/ e (t)) + {q(t),J28<T*(t))jdt, 

which proves (|3ip . □ 

Now we are able to give the following lemma, which is of great importance. 
Lemma 5.8. Let assumptions (A1)-(A4) hold. For any e > and v G L A (S;M. k ), define 

u e (t) = u(t) + ev(t) /or Vt € [0,T]. 

Then we have 

J(u e ) - J{u) =E f (-H(t,x(t),u e (t),p(t),q(t))) - (-H(t,x(t),u(t),p(t),q(t)))dt + o(e) 
Jo 

Proof. According to Lemma 15.61 we have 

J(u e ) - J{u) =E{h x (x(T)),y e (T)) + E F {(f x (t, x(t), u(t)), y e (t)) + 5f(t)} dt + o(e) 

Jo 

=E(-p(T),y*(T))+E [ {(f x (t,x(t),u(t)),y*(t))+5f(t)}dt + o(e). 

Jo 

Applying (f5Tj) . we obtain 
J{u € ) - J{u) 

= -E F {(p(t),5b(t)) + tr[q(t) T 5a(t)} - 5f(t)} dt + o(e) 
Jo 

T 



E / {-H(t, x(t),u e (t),p(t),q(t))) - (-H(t, x(t),u{t),p(t), q(t)))dt + o(e) 
Jo 



□ 



6 Proof of the Main Theorems 
6.1 Proof of Theorem ED 

We follow the technique developed in [8]. Given an optimal 4-tuple (x,u,p,q), define a 
functional H u : L 4 (S;R k ) ^ M as following 

n a (u)=E [ -H{t,x{t),u(t),p{t),q{t))dt. 
Jo 
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By a similar argument as in Lemma f5.4l it can be proved that the functional % u is also locally 
Lipschitz on L 4 (S;R k ). Next, we define Clarke's generalized gradient of the functionals J 
and Ti u at u and explore their properties. 

Definition 6.1. Let Lz{S\R k ) denote the dual space of L 4 (S;R k ) and (•,•) denote the du- 
ality pairing between L 4 (S;R k ) and L~5(S;M. k ). Given an admissible control u E L 4 (S;R k ), 

4 , 

Clarke's generalized gradient of J at u, denoted by dJ{u), is the set of all £ E L3[S;M. ) 
satisfying 

J(u + ev) — J(u) . , . 

J°(u;v) = limsup — - —>(v,(), (32) 

u— >u,e— >0 e 

for all v E L 4 (S;R k ). Clarke's generalized gradient ofH u at u is defined similarly. 

Then, according to Lemma 15,8} given u E L 4 (S;R k ), for any e > and v E L 4 (S;R k ) 
such that u + ev £ L 4 (S;M. k ), we have 

J(u + ev) - J(u) = n a (u + ev) - U a (u) + o(e). 

Hence, we have 

J°{u;v) = (H a )°(u;v), /or W £ L 4 {S;R k ). 
Therefore, by Definition 16.11 we conclude 

dj(u) = dre(u). 

Since u is an optimal control on U ac i, according to Theorem 13.101 

e d J(u) + iV Wad (iZ) = + N Uad (u). (33) 

To characterize Clarke's tangent cone in the L 4 (S;R k ) space, we recall [U Theorem 
8.5.1]. Let (Q,S,fi) be a complete u-finite measure space and X be a separable Banach 
space. Consider a measurable set- valued map K : O X. We associate with it the subset 
K, C L p (Q;X,fi) of selections defined by 

/C := {x £ L p (r2;X, /i)| /or almost all lo £ f2,x(u;) £ i^(o;)}. 

Theorem 6.2. Assume that the set-valued map K is measurable and has closed images. 
Then for every x £ K,, the set valued map u — > T^^(x{oj)) is measurable. Furthermore 

{v £ L p (n-X,n)\ for almost all oj,v(uj) £ T b K{u)) {x(uj))} C T%(x). 

Returning to our proof, since U is convex, by definition, U a( i is also a convex subset of 
L 4 (S;R k ). Therefore, by Theorem 13.81 and Theorem 16.21 we obtain 

Tu ad (u) D {v £ L 4 (S;R k )\v(uj,t) £ T v (u{uj,t)) ^-almost surely}. (34) 

4 , 

The optimality condition ([33]) together with (f34"|) implies that 3( £ L3(S;R ) such that 

£ I (((t), v(t))dt<0 for Vv £ L 4 (S; R k ) such that 
Jo 

< v(t) £ Tu(u(t)) for every t £ [0,T],F-almost surely (35) 

(H u )°(u; v) + E [ (((t),v(t))dt > for Vv £ L 4 (S; M fc ). 
Jo 

Now, we recall a version of the measurable selection theorem in [1]. 
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Definition 6.3. [1, Definition 8.1.2] Let (fl,A) be a measurable space and X be a complete 
separable metric space. Consider a set-valued map F : Q -w X . A measurable map f : $7 h-> X 
satisfying 

Vw G n, f(u) G F(u) 
is called a measurable selection of F. 

Theorem 6.4. fH Theorem 8.1.3] Let X be a complete separable metric space, (0,«4) a 
measurable space, F a measurable set-valued map from to closed nonempty subsets of X. 
Then there exists a measurable selection of F. 

Return to our problem. Fix u G U a( i and (oj, t) G S. Let Q + denote the set of all strictly 
positive rationals. Following the argument in [IJ Page 325] , we have 

T U (u(u,t))=T>(u(u,,t))=f}ci(\j n u ~T ,t) + k B 

n>0 \a€Q+ he[0,a]nQ + ' 

where B denotes the unit ball centred at 0. By [1] Theorem 8.2.4], we conclude that the 
set-valued function Tu{u) is measurable. 

For the first inequality in (|5o]). let M > and define B M = {v G M k : \\v\\ < M}. For 
any positive integer n, define a set- valued function Q^f as follows 



nfM) = < 



{0}, if (C(w, t), u) < — , Vn G S M n ^(€(07, t)) 

1 

{f G Bju D Tii(u(pJ,t)) : (((co,t),v) > — }, otherwise. 

n 



The map — > ((^(uj,t),v} is continuous in n. Moreover, since R fe is separable, 
the map can be expressed as the upper limit of a countable family of measurable functions 
and therefore is measurable. Therefore H^f is measurable since countable intersection of 
measurable set-valued functions is still measurable. Hence, by Theorem 16.41 H^f admits a 
measurable selection v^f G L A (S;M. k ). Note that ([35]) implies that the set 

must have \i measure 0. Hence, we conclude that there exists a set, denoted as S?f , where 

S? = {(u,,t):H*f(u,,t) = {0}} 
and n{S^f) = 1. Consequently, we have 

(C(w, t),v) < - Vf G Bm H Ib(u(o;, t)) on #f . (36) 
n 

Define S M = f|n=i S n with MO = 1 since v( S n) = 1 Vn G N. Moreover, since (J36J) 
holds for all n, we have 

(C(w, t), v) <0Vt)G%fl Ib(u(w, *)) on 5 M . (37) 

Since ([3Tjl holds for arbitrary M, we obtain that 

(C(u,t),v) < for\/v G T(7(-u(w,i)) ^-almost surely. (38) 
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Next, we consider the second inequality in (|35p . Define the partial generalized directional 
derivative of the Hamiltonian H at u(t) in the direction v(t) as 

-H°(u(t); v(t)) = limsup ~ H ^ ^' u ^ ± g(*)) + H(t, x(t),u(t),p(t), q(t)) _ 

Using Fatou's Lemma on the second inequality in (|35|) . we have 

e \ I* -H° u (u{t);v{t)) + (((t),v(t))dt) > (H a )°(u;v)+E C \((t),v(t))dt > (39) 



Let M > and define Bm — {v € F, k : \\v \\ < M}. For any n G N, define a set-valued 
function r„ as follows 



{0}, i/ -flS(u(o; } t);t;) + (C(a;,t),t;) > -- Vt> 6 B A/ 

{w € Sjvf : —H°(u(oj,t))v) + (C(io,t),v) < }, otherwise. 

n 



Using a similar argument as above, with the help of Theorem 16.41 and (|39[) . we can show that 
the set {(ui,t) : T^(oj,t) ^ {0}} must have \i measure 0, which implies that 

- H°(u(u, t); v)) + (C(w, t),v) > ^-almost surely. (40) 

Combining ([38]) and ((IT?]) , we conclude 

£ d u (—H)(t, x(t),u(t),p(t), q(t)) + Nu(u(t)), a.e.t E [0,T], F-a.s. 

6.2 Proof of Theorem 14721 

Given admissible pair (#,«), define 

H(t,x(t),u(t)) = H(t,x(t),u(t),p(t),q(t)) forVte [0,T], F-a.s. 

Under the convexity condition, Clarke's generalized gradient and normal cone coincide with 
the subdifierential and normal cone in the sense of convex analysis. Moreover, combining 
((?]) and the concavity of H(t,x(t), •) for all t G [0, T] a.s, we conclude that 

H(t,x(t),u(t)) = max H(t,x(t),u), a.e. t £ [0,T], F-a.s. 

Define £(t) = x(t) — x(t) satisfying 

d£(t) = {b(t, x(t),u(t)) - b(t, x(t),u(t))} dt 

m 

+ ^2{ai(t,x(t),u(t))-aJ(t,x(t),u(t))}dWJ(t), te[0,T], 

=o. 

Following a standard separating hyperplane argument in convex analysis (see (211 Chapter 
5]), we obtain 

[ T {H(t, x(t),u(t)) - H(t, x(t), «(*))} < [ T (H x (t, x(t),u(t)),at))dt (41) 
Jo Jo 

for any admissible pair (x,u). Detailed proof of (|4ip can be found in 
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Applying Ito's formula to (p(t),£(t)), noting the convexity of h, the inequality (pfT]) and 
the definition of the Hamilitonian (0]), we have 



E{h(x(T)) — h{x(T))} 
>E(h x {x{T%t(T))) 

=E [ {{H x (t,x(t),u(t)),£(t)) - {p(t),b(t,x(t),u(t)) - b(t,x(t),u(t))) 
Jo 

m 

- J2(<lj(t),<r j (t, x(t),u(t)) - a J (t, x(t),u(t)))}dt 

3=1 

>-E I {f(t, x(t),u(t)) - f(t, x(t),u(t))}dt. 
Jo 

Therefore J(u) < J{u) for all u € U a d- 

7 Conclusion 

We have proved a weak version of the necessary and sufficient stochastic maximum principle. 
Instead of insisting on the maximality condition of the Hamiltonian, we showed that belongs 
to the sum of Clarke's generalized gradient of H and Clarke's normal cone at the optimal 
control u. Under certain concavity conditions on the Hamiltonian and objective functions, 
the necessary condition becomes sufficient. The theorem does not involve any second order 
terms, hence no second order differentiability of the coefficients and objective functions is 
required. Moreover, the absence of second order adjoint equation considerably simplifies 
the theorem. Future research on this topic includes the extension of the theorem to more 
general stochastic control systems such as nonconvex control constraints and local Lipschitz 
coefficients. We are currently working on these problems. 

Acknowledgment. The authors are grateful to Professor Nicole El Karoui for the use- 
ful discussions on the paper, especially on the contents of the measurability of stochastic 
processes. 
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